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^^' Abstract. In this paper we consider an analog of the regions of instability 

C^ , for twist maps in the context of area preserving diffeomorphisms which are 

not twist maps. Several properties analogous to those of classical regions of 

instability are proved. 

o 
en 



1. Introduction and Notation 



The objective towards which this article is directed is a good topological picture 
of the dynamics of C" generic area preserving diffeomorphisms of surfaces. We 
Cy ■ address the case where the surface is either the annulus or 5^ and r > 3. Even 

in this case our objective is an ambitious one. Despite some substantial progress, 
many fundamental questions remain unanswered. Perhaps the most important is 
the question of whether there is a C^ generic set of area preserving diffeomorphisms 
^ ' for which the periodic points are dense. It is not difficult to see that for generic 

Os] , diffeomorphisms this question is equivalent to the question of whether the union of 

'^ ' all homoclinic points for all hyperbolic periodic orbits forms a dense set. 

One of our main results, Theorem 9.15, asserts that for any r > 3 there is 
a C generic set of diffeomorphisms of S^ with the property that the union of 
Q>^ [ stable manifolds of all hyperbolic periodic points is a dense set. Of course, one 

^\ • can simultaneously have the union of all unstable manifolds of hyperbolic periodic 

points dense, but this does not imply the desired result about homoclinic points. 
A guiding principle, in pursuing a topological understanding of area preserving 

a diffeomorphisms has been to establish analogs of the classical topological results 

about twist maps (primarily due to G. D. Birkhoff) in a context where the twist 
^ , hypothesis is removed, but generic properties are assumed. 

r '"! ' Our most successful result in this direction is the formulation of a generalization 

rS I of Birkhoff 's notion of "region of instability" and the proof that many of their key 

j^ ' properties remain true for generic area preserving diffeomorphisms of the annulus 

or S'^. Our precise definition of region of instability for an annulus diffeomorphism 
is given in Definition 8.5, but in particular it is an essential open invariant annulus 
which contains no essential compact connected invariant sets which are periodic 
point free. In this sense it is very like the Birkhoff definition for a region of instability 
for a twist map which would require that it contain no essential invariant curves. 
For a generic twist map the two definitions co-incide. 

We show that inside any region of instability there is a hyperperbolic periodic 
point z with the property that K{z), the closure of W^{z), is essential (in the 
annulus) and contains both components of the frontier of the region of instability. 
There are many such "essential" periodic points and we show in Theorem 8.6 and 
Corollary 8.7 that K{z) is independent of the essential periodic point z and that 
we could have equally well defined it to be the closure of the unstable manifold 
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T4^"(z). Further study of the dynamics on K{z) is crucial for a good topological 
understanding of area preserving diffeomorphisms. In particular if, in the generic 
setting, honioclinic points of sucha point z were always dense in K{z) this would 
answer affirmatively the question of whether periodic points are generically dense 
for area preserving diffeomorphisms of the annulus. 

One of the most striking results we establish for regions of instability is a gen- 
eralization of known results (see Mather [M2]) for connecting orbits of twist maps. 
In Theorem 8.11 we show that in a region of instability there are always connect- 
ing orbits between the two components of its frontier. More precisely if the two 
components of the frontier are A and B, then there is an orbit whose a-limit set 
is in A and whose w-limit set is in B, as well as one whose a-limit set is in B and 
whose w-limit set is in A. While Mather's proof of the analogous result for twist 
maps uses variational techniques, he has also made seminal contributions to the 
topological methods on which our work is based (see [Ml], [M3]). 

In the subsequent text, we write T^ for the one dimensional torus and A = T^ x M 
for the infinite annulus. We note N (resp. S) the upper (resp. lower) end of A and 
A the end compactification of A which is a topological sphere, i.e., a topological set 
homcomorphic to the two dimensional sphere 

S^ = {(xi, xa, 2:3) e M^ I xl+xl+xl = 1}. 

We consider the covering projection: 

TT : R2 ^ A 

(x, y) 1-^ {x + Z, y) 

of the universal covering space M^ and the Deck transformation 

r : M^ ^ R2 

{x,y) H^ (x + l,y). 

We give the same name to the first projection 

Pi ■ {x,y) ^ X 

defined on the plane or the annulus. 

A set X of A is called an essential continuum if it is compact, connected, and if 
its complement has two unbounded components. We write U]^ for the upper one 
and Uj^ for the lower one. The sets 

t/+ = [/+ U {N} and Uj^ = Uj^U {S} 

are simply connected domains of A. li A\ K has no other component we will say 
that X is a filled essential continuum. 

We say that a homeomorphism of A preserves area if its preserves the measure 
given by the volume form dx A dy. 

An open set t/ of A is called an essential annulus if it is homcomorphic to A and 
if the inclusion i : U ^^ A induces a one to one map 

i, : Hi{U,R) ^ HiiA,R) 

on the first homology group. If U is bounded, its frontier has two connected com- 
ponents, we denote d'^U (resp. d^U) the upper (resp. the lower one). 
We will consider 

S^ ^{zeC I |z| = l} 
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and 

]D) = {zeC I \z\ <l}. 

2. Prime end theory 

Let U he a non trivial simply connected domain of a topological sphere E (that 
means that the complement has at least two points). We can define the prime end 
compactification of U, introduced by Caratheodory [C], by adding the circle S^. A 
very good exposition of the the theory of prime ends in modern terminology and in 
the context we need can be found in the paper [M3] of John Mather. This paper 
also extends the theory to surfaces other than the plane or 5^ though we shall not 
make use of this. 

The prime end compactification can be defined purely topologically but has 
another significance if we put a complex structure on E. We can find a conformal 
map h between U and the open disk D and we put on UU S^ the topology (which, 
up to homeomorphism of the resulting space, is independent of h) induced from the 
natural topology of D by the bijection 

h:UUS^ ->D 

equal to h on U and to the identity on S^. 

If _F is a homeomorphism of E which leaves U invariant, then the map Fijj 
extends to a homeomorphism of the prime end compactification U \J S^ of U. If 
F\i/ is orientation preserving then its restriction to S^ preserves orientation and we 
can define its rotation number which is an element of T^ . We denote it by pu and 
call it the rotation number of U. 

Every homeomorphism _F of A homotopic to the identity extends naturally to 
an orientation preserving homeomorphism F of A which fixes the ends N and S. 
If K is an invariant essential continuum we write —p]^ (resp. p^) for the rotation 
number of U^ (resp. t/^) defined by F. 

Every lift / of F to the plane induces naturally a lift of Fijj+ to the universal 
covering space. This lift can be extended by a lift (p of <& to the universal covering 
space of S^ which admits a real rotation number pj- such that pj + Z = pj^. In 
the same way we can define p^. 

By prime ends theory we can compactify any invariant and bounded essential 
open annulus, by adding two copies C"*" and C" of S^ corresponding respectively 
to the upper and to the lower end of U. We will get naturally two rotation numbers 
p^ and p^ and for a lift / two real rotation numbers p^ and p^ 

We will often make use of the following classical result about prime ends: 

Proposition 2.1. Let F be an orientation preserving homeomorphism of a topolog- 
ical sphere E and U a non trivial simply connected invariant domain. We suppose 
that: 

i) the rotation number pjj = p/q -^"L is rational, and 

ii) there is a neighborhood of dU in U which doesn't contain the positive or the 
negative orbit of any wandering open set. 

Then dU contains a fixed point of F'' . There is an analogous result for the case 
that U is an open annulus. 
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Proof. Taking F"^ instead of F, we can suppose that the rotation set pu is equal to 
zero. It follows that there is a fixed point 2; G S"^ of the extended homeomorphism 
in the prime end compactification of U. 

From prime end theory we can find a sequence of open arcs (7n)^i in U of diam- 
eter < 1/n each of which extends to a closed arc in the prime end-compactification, 
with end points which belong to S^ and which are close to z, one on each side. 
Moreover, these arcs have the property that the closures 7„ converge to the point 
z in the prime end compactification as n goes to infinity. 

Each of these arcs divides U into two simply connected domains. We denote by 
Un the small one (i.e. z belongs to the closure of t/„ in the prime end compactifi- 
cation). We wish to show that -F'(7ra) n7„ ^ 0. If the arc 7„ doesn't meet its image 
under F, the fact that z is fixed implies that one of the inclusions F{Un) C Un 
or F~^{Un) C Un is true. In the first case we can find an open set V contained 
in Un \ F{Un) which is wandering and has a positive orbit contained in C/„. In 
the second case we can construct a wandering domain whose negative orbit is con- 
tained in Un- Since these conditions contradict our hypothesis we conclude that 

F{-in) n 7n 7^ 0. 

This fact together with the fact that the diameters of the arcs 7„ tend to zero 
implies that any limit point in dU is a fixed point of F. D 



Remark 2.2. We note that in the proof above we showed that for any fixed prime 
end z and any sequence of arcs 7„ defining it, we have F{"/n) H 7ri =/= 0. We shall 
make use of this fact subsequently. 

We deduce immediately the following corollaries of Proposition 2.1: 

Corollary 2.3. Let F be an orientation preserving homeomorphism of a topological 
sphere S and U a non trivial simply connected invariant domain. We suppose that: 

i) the rotation number pjj =^ p/q + 1, is rational, and 
ii) there is no wandering point in U . 

Then dU contains a fixed point of F'^ . 

Corollary 2.4. Let F be an area preserving homeomorphism of A homotopic to 
the identity and K an invariant continuum. We suppose that p^ = p/q + 7L (resp. 
p^ = p/q + Z) is rational. Then K contains a fixed point of F"^ , more precisely the 
fixed point is in dU]^ (resp. dU^). 



3. Rotation number of invariant measures 

Let F be a homeomorphism of A homotopic to the identity and / a lift of F to 
M^. If /x is an invariant Borel probability measure we can define its rotation number 

p{l^) == / <Pdp,, 

where (/> : A ^ M is the function lifted by the function pio f ~ pi- 

If O is a periodic orbit of F with q elements, there exist an integer p £ Z such 
that f^iz) — TP{z) for every z £ 7r^^(0). We will say that O is a periodic orbit of 



REGIONS OF INSTABILITY FOR NON-TWIST MAPS 5 

type {p,q). The number p/q is called the rotation number of O, it is equal to the 
rotation number of the invariant measure 



n ^ — ^ 



q 

^ zeo 

where Sz is the Dirac measure concentrated on z. 

If ii" is a non empty compact invariant set, the set of Borel probability measures 
whose support is in K, is a compact set for the weak topology and the subset M{K) 
of the invariant measures is a non empty convex and closed set. We can define the 
rotation set of K 

p{K) = {p{^i) \ fieMK}, 

which is a non-empty segment [a, /3] of M. We have the following classical result: 

Proposition 3.1. For every e > there exists N > such that for every z G 
'K^'^(K) and for every n > N we have: 

Piof"{z)-pi{z) 

a — e < < p + e. 

n 

Proof. Let's suppose for example that we can find a real number e > 0, a sequence 
of integers {nk)k>o converging to +oo and a sequence {zk)k>a in '!t^^{K) such that 

pio f"''{zk) -pi{zk) 



Tik 



>(3 + £. 



We have the inequality 



where 



4) d^k > P + £ 



Hk ^ ''" 

1=0 



Any closure value ^ of the sequence {pk)k>a belongs to M{K) and satisfies the 
inequality 



(I) dp > (3 + e. 

We have got a contradiction. D 

We obtain the following corollary which applies for example when K is an in- 
variant essential simple closed curve. 

Corollary 3.2. If p{K) = {a} the sequence 

Pi ° ./" - Pi 
n 
converges uniformly on Tr^^(K) to the constant function a. 

In the same way we can define the rotation number of an invariant probability 
measure when i^ is a homeomorphism of the closed annulus T^ x [0, 1] homotopic to 
the identity, for a lift / to M x [0, 1]. The rotation set of / is the set of the rotation 
numbers of all the invariant measures. It is a segment which contains the rotation 
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number of the two boundary curves of the annulus. We wiU use the foUowing result 
proved in [Fl]: 

Theorem 3.3. Let F be an area preserving homeomorphism of the closed annulus 
T^ X [0,1] homotopic to the identity and f a lift of F to R x [0,1]. For every 
rational number p contained in the rotation set of f , there exists a periodic orbit of 
type {p, q), where p = p/ q and p and q are relatively prime. 

4. Indices 

Let F be an orientation preserving homeomorphism of a topological sphere S 
and D a closed disk of S (i.e. a set homeomorphic to D). If there is no fixed point 
of F in 9-D, we can define the index i{F, D) of D (see [D] for example). This index 
is equal to the sum of the Lefschetz indices i{F, z) of the fixed points z contained 
in D if there are a finite number of fixed points. 

If we can find a finite family of disjoint closed disks {Dk)i<k<n such that every 
fixed point of F is contained in the interior of one these disks, we have the Lefschetz 
formula: 



fe=i 
If C/ is a non trivial simply connected domain of S with a non zero rotation 
number, we can find a closed disk D <Z U whose interior contains all the fixed 
points in U and such that i{F, D) = 1. We deduce that U contains at least one 
fixed point. 

5. Invariant essential continua and periodic orbits. 

We begin with a useful extension lemma 

Lemma 5.1. Let F be a homeomorphism of A homotopic to the identity and f a 
lift of F to M.^ . We suppose that K is an invariant essential continuum and that 
Pk -^ *-*■ Then there exists a homeomorphism F' of A homotopic to the identity 
and an essential simple closed curve C^ C U^ such that: 

i) the map F' coincides with F on the complement of a closed set contained in 

ii) the curve C^ is invariant by F' and its rotation number is > (for the lift 

f which coincides with f on tt^^{K)); 
iii) the invariant set U]^ D C/^+ of F' contains no fixed points of rotation number 

0; 
iv) the map F' is area-preserving if F is. 

Proof. We can find a complex structure on A which defines the same differential 
structure on A. Let's consider a conformal map h between the simply connected 
domain W^ and P and the maps 

r:C/+HO,l[ 

z H^ 1^(^)1 



REGIONS OF INSTABILITY FOR NON-TWIST MAPS 

and 



\hiz)\ 



We consider also a lift 

of 9 to the universal covering space and the map 

f:7r-i(C/+)H0,l[ 

given by r{z) — r(7r(z)). 

For every t e]0, 1[ we consider the essential simple closed curve Ct = r^^{{t}) of 
U^. Let's fix to < 1/2 such that the curve (7+ = Ctg is above the image of C1/2. 
We can find a homeomorphism between the closed annulus Aq delimited by C1/2 
and C'tf, and the closed annulus Ai delimited by F(Ci/2) and Ctg, which is equal to 
F on Ci/2 and to the identity on Ctg ■ We denote by Fi its extension to the infinite 
annulus which is equal to F below C1/2 and to the identity above Ctg ■ If F is area 
preserving the fact that K is invariant tells us that the two annuli Aq and Ai have 
the same area and we can choose the homeomorphism Fi to be area-preserving (so 
is the map Fi). 

We write /i for the lift of Fi which coincides with / on Tr^^{K). Using the 
condition p^ > 0, wc can find e e]0, l/4[ such that 

f(z)>l-2e => 0ofi{z)>e{z). 
Using an argument of compactness, we can find M > such that 

r{z) >to => 9o fi{z) < e{z) + M. 

Consider a C°° decreasing map 

A:]0, IH [0,+oo[ 

equal to on [1 — e, 1[ and let's define the map F2 equal outside of V^ to the 
identity and on U^ to the time one map of the Hamiltonian flow of A o r for the 
usual symplectic structure on A. That means the flow defined by the equations: 

dv 

x' = \'{r{x,y)) — {x,y) 
dy 

dv 

y' = -\'{r{x,y))-—{x,y). 
dy 

The map F' — F10F2 satisfies the conditions i) and iv) of the lemma. It also fixes 
the curve C'^ . We can choose the function A such that its satisfies the condition 
Hi) and the second part of the condition ii). 

Indeed, if /2 is the lift of F2 equal to identity on t:^^{K), we have 

d°f2{z)>9{z) and r o f2{z) — r{z) 
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for every z G 7r^^(C/^), so we have the relation 

r{z) > 1 - 2e ^ 'do f{z) > 9{z). 
If we choose the function |A'| sufficiently large on [ro, I — 2e] we will get 

f(z)e[ro,l-2£] ^ eof'{z)>e{z). 

n 

We have a similar result in the cases where pj < 0, p^ > and p^ < 0. 
The main result of this paragraph is the following. 

Proposition 5.2. Let F be an area preserving homeomorphism of A homotopic to 
the identity and f a lift of F toM."^ . If K is an invariant essential continuum which 
doesn't contain any periodic orbit of F , then: 
i) the set K is a filled essential continuum; 

ii) there exists a G R \ Q such that p{K) — {a\; 

iii) Pk = Pk = "• 

Proof. We first prove assertion i) using the results of the paragraphs 2 and 4. 
Suppose that A\ K has a bounded component Uq. It is periodic and we can 
suppose it fixed by taking an iterate of F instead of F. Consider the extension F 
of F to the end compactification A. 

If U is any invariant bounded component oi A\ K then it is topologically an 
open disk and must contain a fixed point since F is area preserving. The fact 
that K doesn't contain any fixed points implies that the set of invariant bounded 
components of A \ ii' is finite, since otherwise there would be a sequence of fixed 
points with limit in K . Moreover we can find a finite family (-Dfc)i<fc<n of closed 
disks, one in each fixed component oi A\ K , such that every fixed point of F is 
contained in the interior of one these disks and such that i(F,Dk) = 1 for every 
k. There arc at least three components: U"^, U]^ and [/q- This contradicts the 
Lefschetz formula. 

We next prove assertions ii) and iii). Using Proposition 2.1, we know that the 
rotation numbers pj- and p^ are irrational. Let's consider a rational number p/ q. 
Applying the extension lemma to F'^ and to the lift /^ oT"^, one can construct an 
area preserving homeomorphism F' of A which coincides with F'^ on a neighborhood 
of K and which fixes two essential simple curves C+ C U^ and C^ C C/^ such 
that: 



there is no fixed point of rotation number in t/^ fl [/„+ ; 



• there is no fixed point of rotation number in Uj^ n t/^_ ; 

• the rotation number of C^ and qp^ — p have the same sign; 
the rotation number of C^ and q])^ — p have the same sign. 



• 



Using the fact that K is filled and doesn't contain any periodic orbit we deduce 
that the map F' has no fixed point of rotation number in the closed annulus 
defined by the curves C"*" and C~ . From Theorem 3.3 we deduce that: 

• the rotation set p{K) doesn't contain p/q; 
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• the numbers p^ and p^ are not separated by p/q. 

The conclusion follows immediately. D 

Definition 5.3. Such a set will be called an irrational invariant essential contin- 
uum and a = p{K) its rotation number. 

In the same way, we can prove the following result. 

Proposition 5.4. Let F be an area preserving homeomorphism of A homotopic to 
the identity, f a lift of F to R^ and K an invariant essential continuum. For every 
rational number p contained in p{K), there exists a periodic orbit of type {p,q) 
contained in K , where p = p/q and p and q are relatively prime. 

Proof. We give a proof by contradicting the assumption that F has no periodic 
points of type (p, q) in K. If this is assumed then F has no periodic points of type 
(p, q) in some neighborhood Vq oi K since the periodic points of type (p, q) form a 
closed set. It then follows from Proposition 2.1 that p^ ^ p/q and p^ ^ p/q. 

Using Lemma 5.1 we construct an area preserving map F' ^ which coincides with 
i^' on some neighborhood 1/ C Vb of K, which leaves invariant two essential simple 
closed curves C+ C U^ and C^ C U]^ , and which has no fixed points of rotation 
in 7r~^(F) for the lift equal to f^ oT^p on tt~^{K). By Theorem 3.3, there exists a 
fixed point of F' of rotation number in the annulus B delimited by C+ and C~ . 
Since it is not in K and not in V this fixed point must lie in a bounded invariant 
component oi B \ K and hence a bounded invariant component oi A\ K. 

Let p : Bq ^ B he a finite covering map of degree greater than the rotation 
number of any periodic point of F' . Then there is a lift G : Bq ^ B^ oi F' such 
that z e So is a fixed point of G if and only if p{z) is a fixed point of F' of rotation 
number zero. Let K^) — p~^{K) so that G has no fixed points in Kq. Since any 
component of B \ K containing a fixed point of F' of rotation number zero is an 
invariant open topological disk it follows that any component of Bq \ Kq containing 
a fixed point of G is also an invariant open topological disk. 

As in the proof of Proposition 5.2, the fact that Kq and the boundary of Bq 
don't contain any fixed points of G implies that the set of bounded components 
of Bq \ Kq containing fixed points is finite. Moreover we can find a finite family 
{Dk)i<k<n of closed disks, one in each component of Bq\Kq containing such a fixed 
point, such that every fixed point of G is contained in the interior of one these disks 
and such that i{G, D^) — 1 for every k. There is at least one such disk since there 
was at least one fixed point of F' of rotation number zero. This again contradicts 
the Lefschctz formula for G : Bo ^ So- CH 



6. Generic properties of diffeomorphisms of the sphere. 

We consider a smooth surface S diffeomorphic to 5^ and a volume form uj on it. 
For every integer fc > 1 we write Diff^(S) for the set of C" diffeomorphisms which 
preserves to. We put on that set the natural C-topology. If z is a periodic point 
of period q oi F E Diff^(S), we have three possibilities: 

• the eigenvalues of DF'^{z) are both equal to 1 or —1: the point z is degenerate: 

• the eigenvalues of DF{z) are two different conjugate complex numbers of 
modulus 1: the point z is elliptic] 
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• the eigenvalues A and /i of DF{z) are real and satisfy |A| < 1 < |/i|: the point 
z is hyperbolic or a saddle point. 

The stable (resp. unstable) branches of a hyperbolic periodic point z are the 
two components of W^{z) \ {z} (resp. W^{z) \ {z}) for the topology induced by a 
parameterization of W'^{z) (resp. W'^{z)). Let's recall that a saddle connection is a 
branch F which is a stable branch of a hyperbolic periodic point z and an unstable 
branch of a hyperbolic periodic point z' (perhaps equal to z). 

Lemma 6.1. Suppose F is an area preserving dijfeomorphism of a connected open 
subset U of S^ and K C U is a compact connected F -invariant set. If T is a branch 
of a hyperbolic periodic point of F then either T d K or T is entirely contained in 
a single component of U \K. 

Proof. Suppose F has non-empty intersection with both K and U \K. Let Uq be 
a component oi U \K which contains a point of F. Then Uq is a periodic domain 
since every component of the complement oi U \ K is. There is an an open arc 
a C F n t/o whose endpoints arc in K . 

We note that if z > 0, then F''-{a) and a are disjoint since F*(a) cannot be a 
subset of a (as a is in a branch) and F^{a) cannot contain an endpoint of a. 

Let V be the component oi S'^\K which contains Uq. Since V is simply connected 
a separates it into two non-empty simply connected domains Vi and V2 each of 
which has non-empty intersection with Uq. It is not difficult to show that this 
implies a separates C/q into two connected open sets C/i = C/q H Vi and U2 = t/o H V2 • 

Choose A: > so that F^{Uo) = Uq. The map F is area preserving, so Ui x U2 
is a non wandering open set for F^ x F^ . We deduce that there exists an integer 
n > 1 such that F"'=(t/i) n t/i 7^ and F"''{U2) n C/2 7^ 0- Moreover the closure in 
Uq of F'^^{Ui) is not contained in C/i, it meets the boundary a. Similarly a meets 
the closure of F"'^(f72) in f/o- We deduce that a meets its image by F^^ , which 
is impossible. We conclude it is not possible for F to have non-empty intersection 
with both K and its complement. If F lies in the complement of K it must be in a 
single component because it is connected. D 

We will say that a periodic point z of period q is Moser stable if it admits a 
fundamental system of neighborhoods which are closed disks D such that Fqo is 
minimal. 

We next present the following result of Mather [Ml]. 

Theorem 6.2. (Mather) Let F G Diff^(S) such that: 

• there is no degenerate periodic point, 

• every elliptic periodic point is Moser stable, and 

• there are no saddle connections. 

Then, we know that: 

i) the rotation number of any non trivial simply connected periodic domain is 

irrational; 
ii) the branches of a hyperbolic periodic point have the same closure. 

Proof. We first will prove assertion i) by contradiction. If assertion i) is false then 
by replacing F with an iterate we can assume that there is an invariant simply 
connected domain U with rotation number zero. This implies that the extension of 
F to the prime end compactification of U has a fixed point z on the boundary. 
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From the theory of prime ends and the proof of Proposition 2.1 we know that 
there is a sequence of open arcs {"fn)'^=i in U , each of which is the interior of a 
closed arc with endpoints in dU ^ which define the prime end z. Moreover, we can 
assume the sequence of diameters of (7„) tends to zero as n tends to infinity, so by 
choosing a subsequence we can assume the closures of the arcs converge to a point 
p in dU. We observed (see Remark 2.2) that i^(7n) H 7„ 7^ for each n and hence 
p is a fixed point of F . 

We note first that this fixed point p cannot be a Moser stable point. This is 
because a Moser stable point is in the interior of an arbitrarily small F invariant 
disk on the boundary of which F is minimal. Thus the boundary of such a disk 
must intersect the complement of U and hence, by minimality, the boundary is 
disjoint from U . This would imply that the entire disk is disjoint from U which is 
impossible. 

It follows that the point p must be a hyperbolic fixed point. By the Hartman- 
Grobman theorem there is a neighborhood V of p on which there are topological 
co-ordinates (x, y) with p at the origin and in which F{x,y) = {2x,y/2). We will 
use these local coordinates on V and refer to the four local branches at p by which 
we mean the intersection of V with the positive and negative x and y axes. 

The contradiction we wish to derive will follow from the following lemma 

Lemma 6.3. Replacing the sequence {7n} with a subsequence, we may conclude 
that there is a neighborhood W <Z V oj p such that one component of U H W 
contains the arcs 7„ for n sufficiently large, and this component consists of one of 
the following: 

a) the intersection of W with a single open quadrant in the {x,y) co-ordinates, 
or 

b) the intersection of W with two adjacent open quadrants and the local branch 
between them, or 

c) the intersection of W with three open quadrants and the two local branches 
between them, or 

d) the complement in W of p and a single local branch. 

Proof. We first observe that if 7 is a local branch oi p uiV then by Lemma 6.1 7 
must be entirely contained in U or disjoint from U . If it is contained in U there is 
an X G 7 such that a neighborhood of the interval [x, F{x)\ is in U . The collection of 
all iterates of this neighborhood contain the intersection of a smaller neighborhood 
Wq with the two open quadrants in (cc, y) co-ordinates of which 7 is part of the 
boundary. Thus if one point of a local branch is in U then, perhaps in a smaller 
neighborhood, the entire branch and the open quadrants on either side of it are in 
U. We shall make repeated use of this property. 

The first use of this fact is to observe that it is not possible for all four local 
branches at p to be in U as this would imply that there is a neighborhood W oi p 
such that W \ {p} C U. This would mean that p G U which is not the case. 

We again use the fact that if a local branch intersects U then (perhaps in a 
smaller neighborhood) the entire branch and the open quadrants on either side of 
it are in U. This implies that either one of properties b) - d) above holds or there 
is a single quadrant such that the interiors of infinitely many of the arcs 7„ are in 
that quadrant, but the two branches which bound this quadrant are not in U . We 
complete the proof of the lemma by showing that this last alternative implies that 
a) holds. 
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We can suppose, without loss of generality, that it is the open first quadrant 

Q^{{x,y)\x>Q, y>0} 

which contains the interiors of infinitely many of the 7„ and the positive x and y 
axes are in the complement of U . By choosing a subsequence we may assume every 
7„ is in the first quadrant. We need only show that there is a neighborhood W d V 
oi p such that W DQ is entirely in U. 

To do this, for each small e > we consider the arc Fg made up of the two line 
segments {{x,y) = (t,e)|0 < t < e} and {{x,y) — {e,t)\0 < t < e}. These two 
segments form two sides of a rectangle i?g the other sides of which lie in the x and 
y axes. This rectangle contains infinitely many of the arcs 7„ and in particular 
points of U. Also if e is sufficiently small we may assume this rectangle is disjoint 
from 7i and in particular cannot contain all oi U . It follows that the arc Fg has 
non-empty intersection with U . 

We want next to show there is an open sub-arc P of Fg with the property that 
it is in U but its endpoints are in the complement of U, and that F(/3) n/3 7^ 0. To 
see this we note that Fg separates 71 and jn for a large N. Hence some component 
/9 of Fg n [/ must separate 71 and jn in U . Clearly /3 is an open sub- arc of Fg with 
endpoints in the complement of U . 

Let Vi and V2 be the components oi U\f3 which contain 71 and jn respectively. 
The fact that ^(71)071 ^ and F(7Ar)n7Ar ^ implies F{Vi)r\V, ^ for i = 1, 2. 
Since F is area preserving F(Vi) cannot be a proper subset of Vi. We conclude that 

F{i3)ni3^9. 

Since /? C Fg and i^(Fg) n Fg consists of the single point {x,y) = {e,e/2) we 
conclude that both {e, e/2) and its inverse image (e/2, e) are in (3. Thus the arc Jg 
made of the two line segments {{x, y) — (t, e)|£/2 <t<e} and {{x, y) — (e, t)|£/2 < 
t < e} is entirely in U . Similarly for any (5 < £ we have Js d U and indeed for 
any n € 1^, F"{Js) C U. This easily implies that i?g D Q C U, which means that 
property a) holds. D 

We next observe that this lemma and the assumption that i) is false do lead to a 
contradiction. This is because from the lemma we see that there is a neighborhood 
of the prime end z corresponding to p in the prime ends of U which consists of 
accessible prime ends corresponding to points in either stable or unstable branches 
of z. It also follows that there are finitely many fixed prime ends and they all have 
this property. We can conclude that any non-fixed prime end e, which is between 
the fixed prime ends zi and Z2 niust correspond to a point on a branch of pi and 
a branch of p2 where pi is the fixed point in dU corresponding to the prime end 
Zi. Since this is true for the entire interval of prime ends between zi and Z2 there 
is a saddle connection between pi and p2 which contradicts our hypothesis. We 
conclude that i) of the theorem must hold. 

Property ii) of the theorem is a consequence of i) and Lemma 6.1. As before, 
by taking an iterate we may assume the point in question, p, is fixed. The closure 
of one branch of this point is a compact connected invariant set K. From Lemma 
6.1 we know that another branch of p is either a subset of K or disjoint from it. It 
cannot be disjoint since if U denotes the component of the complement of K which 
contains 7 then U would be an invariant simply connected domain and the fixed 
point p would be an accessible point of dU. This would mean that the rotation 
number of U is rational and hence property i) would be contradicted. 
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We conclude j d K . Since we have shown any branch of p is a subset of the 
closure of any other, property ii) follows. D 

We remark that for r > 3 the set of diffeoniorphisnis satisfying the hypotheses 
of Theorem 6.2 is generic in Diff^(S). 

We now recall a result of Robinson [R2] and Pixton [P] . It was first proved by 
Robinson in the case of fixed points and generalized by Pixton to periodic points. 

Theorem 6.4. For every r > 1, there exists a generic set Gr in Diff^(S) such 
that for every F in Gr the following properties are satisfied: 

1) if z is a hyperbolic periodic point, any stable branch T of z has non-empty 

transversal intersection with any unstable branch T' of z. Moreover T — V . 
ii) if z and z' are two distinct hyperbolic periodic points such that z' belongs to 
the closure of a branch of z, then any stable (resp. unstable) branch T of 
z meets transversely any unstable (resp. stable) branch V of z' . Moreover 

r = F. 

We make some comments on these theorems. 

• Every periodic connected and compact set which contains a hyperbolic periodic 
point, and which is not a single point, contains all the branches of that point. 

• If we use Mather's theorem with the stronger (and generic) hypothesis that any 
intersections of branches are transverse, then assertion ii) of Mather's theorem can 
be used to derive assertion i) of Pixton's theorem (see also Oliveira [O] or Arnaud 
[A]). Moreover we have the following easy fact: 

• Any hyperbolic periodic point has a fundamental system of neighborhoods which 
are closed disks whose boundary is contained in W{z)UW'^{z). This property and 
the classical A- lemma (see Theorem 11.1 of Robinson [Rl]) implies the condition 
ii) of Pixton's theorem. 

7. Essential and inessential hyperbolic periodic points. 

In this section we consider an area preserving C^-diffeomorphism of S'^ such 
that: 

i) there are no degenerate periodic points, 
ii) every elliptic periodic point is Moser stable, and 

iii) the intersection of any branches of hyperbolic periodic points are transverse 
and any stable and any unstable branch of the same periodic point have non- 
empty intersection. 
If Aq is a connected open subset of S^ whose complement has exactly two com- 
ponents, neither of which is a single point, we will call it an annular domain. It is 
homeomorphic to an open annulus. If Aq is an _F- invariant annular domain then 
we make speak of an essential continuum K C Aq meaning one which separates the 
two components of the complement of Aq. 

Mather's theorem and Pixton's theorem immediately imply the following corol- 
laries. 

Proposition 7.1. For a diffeomorphism F of S^ satisfying i) - iii) above we have 
the following properties: 
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1. every periodic simply connected domain has an irrational prime end rotation 
number; 

2. every periodic annular domain has both prime end rotation numbers irrational; 

3. if Ao is an invariant annular domain and K C Aq is an invariant essential 
continuum, the rotation numbers pj^ and p^ are irrational; 

4. every stable branch F" and every unstable branch T" of a hyperbolic periodic 
point have a non-empty transversal intersection and we have F* ~ F"; 

5. if z is a hyperbolic periodic point and F* and F" are a stable and an unstable 
branch of z then there is a fundamental system of neighborhoods of z which 
are closed disks whose boundary is contained F" U F"; 

6. a periodic compact connected set, which is not a single point, and which con- 
tains a hyperbolic periodic point, contains the branches of that point; 

7. if the closure of a branch of a hyperbolic periodic point z contains a hyper- 
bolic periodic point z' , then every stable (resp. unstable) branch T of z meets 
transversely any unstable (resp. stable) branch F' of z' and we have F = F'. 



Let _F : Aq ^ Aq be an area preserving difFeomorphism of an annular domain 
which extends continuously to clAo C S*^. We consider a hyperbolic periodic point 
z for F. We will assume the generic property that each stable branch of z has non- 
empty intersection with each unstable branch and all intersections are transverse. 

Definition 7.2. // there is an essential closed curve in Aq consisting of a finite 
number of segments of branches of z we will say that z is essential, otherwise we 
say that z is inessential. 

It is immediate that a hyperbolic periodic orbit O consists of points which are 
all essential or all inessential. 

Proposition 7.3. Suppose z €z Aq is a hyperbolic fixed point of F. Let U{z) be the 
union of all closed topological disks in Aq whose boundaries consist of finitely many 
segments of the branches of z, then 

i) U{z) is an open, connected, and F -invariant set which contains z and its 

branches. 
ii) If z is inessential then U{z) is simply connected. 
iii) If z is essential then U{z) is annular and essential in Aq. 
iv) If V C Aq is any other open connected F -invariant set which is either simply 

connected or annular and essential then either U{z) C V, V C U{z), or 

u{z)r]V^9. 

Proof Let z be a hyperbolic fixed point of F and let U{z) be the union of all closed 
topological disks in Aq whose boundaries consist of finitely many segments of the 
branches of z. Clearly F{U{z)) = U(z). Property 3 of Proposition 7.1 implies z is 
in the interior of U{z). Since any closed segment on a branch of z can be mapped 
by a power of F into a small neighborhood of z we see that these segments are in 
the interior of one of the closed disks whose union forms U{z). It follows that any 
of these closed disks is contained in the union of the interiors of finitely many other 
such disks, so U{z) is also equal to the the union of the interiors of these disks and 
is open. This proves i). 

We next show ii). This is equivalent to showing that z inessential implies the 
complement of U{z) in 5*^ is connected. If this is not the case then there is a 
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simple closed curve a in U{z) which separates two components, call them Ki and 
K2, of the complement of U{z). Since U{z) is the union of the interiors of closed 
disks whose boundaries are branch segments, the curve a is covered by the interiors 
of finitely many such disks. Let V denote the union of these open disks and let 
P be the frontier of the component of S*^ \ T^ containing Ki. Then /3 is a simple 
closed curve which separates Ki and K2 and consists of finitely many segments of 
branches of z. If /3 were essential in Ag then z would be essential. Hence we know 
/3 bounds a disk in Aq which must contain one of Ki or K2. This would imply that 
one of Ki or K2 is in U{z) which is a contradiction. Hence we have shown that z 
inessential implies that U{z) is simply connected. 

If z is essential then a similar argument shows that the complement of U{z) has 
exactly two components so in) follows. More precisely, the fact that z is essential 
implies U{z) is not simply connected so S^ \ U{z) has at least two components and 
in particular there are distinct components Ki and K2 each of which contains one 
of the components of S*^ \ Aq. If in) does not hold then the complement of 5^ \ U{z) 
has at least three connected components: Ki and K2 and one more K3 which is 
a subset of Aq. There is a simple closed curve a in U{z) which separates K^ from 
Ki U K2 and hence from S'^ \ Aq. As above we can find /3o a simple closed curve 
which separates K3 and S*^ \ Aq and which consists of finitely many segments of 
branches of z. Since /3o does not separate the two components of S'^ \ Aq it bounds 
a closed disk in Ag. The interior of this disk contains K^ but is also in U{z), a 
contradiction. We conclude that S^ \ U{z) has only two components. 

Finally we show iv). If U{z) does not intersect both V and the complement of 
V then either U{z) C F or U{z) n F = 0. But if U{z) does intersect both V and 
its complement then either V is entirely contained in the interior of a disk whose 
boundary consists of branch segments (in which case V C U{z) ) ovV contains part 
of a branch oi z. liV contains a point of a branch of z then it contains the entire 
branch by Lemma 6.1 and hence all branches of z. Thus if D is any closed disk in 
Aq whose boundary consists of branch segments then the boundary of D \s in V 
and inessential in Ag. This boundary cannot be essential inV asV is either simply 
connected or annular and essential in Ag. Hence the boundary of D is contractible 
in V . Since Z? C Ag we conclude that Z? C F. Since this is true for any D we have 
shown that U{z) C K D 

Proposition 7.4. Suppose z Cz Aq is an essential hyperbolic fixed point of F. Then 
the union of any stable branch and any unstable branch of z contains an essential 
simple closed curve and a point in each component of the complement of this curve. 

Proof. From the definition of essential hyperbolic fixed point we know that there is 
an essential curve /3 in Ag consisting of segments of branches of z which intersect 
transversely. The fact that any stable branch of the fixed point has non-empty 
transverse intersection with any unstable branch allows us to deduce that this 
curve can be chosen to lie in the union of any pair of branches, one stable and one 
unstable. Let ag be the frontier of one of the components of the complement of f3 
which intersects one component of /S'^\Ag and let a be the frontier of the component 
of the complement of ag which contains the other component of 5^ \ Ag. Then a 
consists of segments of branches of z which intersect transversely and is simple. 
Since it separates the two components of S"^ \ Ag it is essential. Since a is in 
the interior of U{z) the branches of z contain points from each component of the 
complement of a. D 
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8. Regions of instability. 

We begin with the foUowing resuh which improves Proposition 5.4 in the generic 
case. 

Proposition 8.1. Suppose F is an area preserving dijfeomorphism of S^ with an 
invariant annular domain Aq and P is the set of periodic points of F with period 
q. Assume 

i) the set P is empty or finite and all points of P are non- degenerate, 
ii) every elliptic periodic point in P is Moser stable, and the invariant, F min- 
imal, Jordan curves bounding neighborhoods of elliptic periodic points have 
rotation numbers which are not constant in any neighborhood of the elliptic 
periodic point, and 
iii) the intersection of any branches of hyperbolic periodic points in P are trans- 
verse and any two branches of the same periodic point have non-empty inter- 
section. 

Let K C Aq be an invariant essential continuum. If p is relatively prime to q and 
P = p/q ^s in p{K), then P is not empty and there exists an essential hyperbolic 
periodic orbit in K C] P with rotation number p. The analogous result holds if 
A = T^ x [0, 1] and F : A ^ A has no periodic points on the boundary of A and 
satisfies i) - iii). 

Proof. Taking F'^ instead of F and f^ o T~p instead of / we can suppose that p = 0. 
Replacing _F : Aq — ^ Aq by a Uft to a finite cover we may assume that there are 
no fixed points except those which project to fixed points with rotation number 0. 
Note that a continuum is essential if and only if a lift to a finite cover is essential. 
Hence it suffices to prove the result for this lift since it has an essential continuum 
which projects to K . By abuse of notation we refer to the lift as i^ : Ag ^ Ag. 

Consider F the extension of F to A the end compactification of Aq, i.e. the 
compactification obtained by collapsing each of the two components of S^ \ Aq to 
a point, obtaining a space A homconiorphic to S"^ . We will denote the two points 
added in this compactification by z+ and z^ . If [/"*" and U~ are the components of 
K\K containing z+ and z^ respectively then they are simply connected _F-invariant 
domains. We can apply Theorem 6.2 to show these domains have irrational rotation 
number despite the fact that at the points z^ and z~ the homeomorphism F is 
not smooth and neither is the invariant measure. The proof of part i) of 6.2 is still 
valid. 

The theory of prime ends allows us to conclude that there is a closed disk 
£)+ C t/"*" which contains all the fixed points of F which are in t/+ and such 
that i{F, £)+) = 1. We can find D^ C U^ with the analogous property. 

By Proposition 7.3 to every inessential hyperbolic fixed point z S Aq, we can 
associate an invariant simply connected domain U{z). The same proposition implies 
there are a finite set of such domains which are disjoint and which contain all 
inessential fixed points of F (all of which, by the remarks above, can be assumed 
to have rotation number zero). Moreover they are either disjoint from the annular 
regions U^ \ {z+} and U^ \ {z^} or contained in them. In each such domain 
U{z) which is disjoint from C/+ U U^ we may choose a closed disk D C U{z) which 
contains all the fixed points in U{z) and such that i{F, D) = 1. We write {I?i}i<i<fe, 
k > 0, for the family of disks obtained in this way. 
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We next consider the finite family {zj}i<j<i, I > 0, of fixed points of F which 
are outside all the disks {Di}. By the Lefschetz formula, we have 

I k 

3=1 i=l 

and we can write 

I 



j=i 



i{F, Zj) + k = 0. 



Proposition 5.4 tells us that one of the integers fc or Ms > 1. We know that 
i{F, Zj) — ±1 for every j and that i{F, Zj) — 1 if Zj is elliptic. We deduce that I > 1 
and that there exists a fixed point Zj such that i{F, Zj) < 0. This point has to be 
hyperbolic. It is essential because it is not contained in the disks Di. 

The analogous result for yl = T^ x [0, 1] follows from similar but easier reasoning. 

D 

Corollary 8.2. Suppose F is an area preserving diffeomorphism of S"^ with an 
invariant annular domain Ag and satisfies the hypothesis of Propositon 8.1 for 
every q. Then every invariant essential continuum K in Aq which is contained in 
the frontier of a connected open set U is irrational. 

Proof. If K is not irrational (see Definition 5.3) it contains a periodic point z. By 
Proposition 8.1 we can suppose that it is an essential hyperbolic periodic point. 
The set K contains an essential simple closed curve C (in W''{z) U W'^{z)), and a 
point in each component of the complement of C by Proposition 7.4. We deduce 
that the open set U meets the two components of A \ C and doesn't meet C. The 
contradiction comes from the connectedness oiU . D 

The results above motivate the following definition. 

Definition 8.3. Suppose F is an area preserving diffeomorphism of an annular 
domain Aq and P{q) is the set of periodic points of F with period q. Assume that 
for all q 

i) the set P{q) is empty or finite and all points of P are non-degenerate, 
ii) every elliptic periodic point in P{q) is Moser stable, and the invariant, F 
minimal, .Jordan curves bounding neighborhoods of an elliptic periodic point z 
have rotation numbers which are not constant in any neighborhood of z, and 
iii) the intersection of any branches of hyperbolic periodic points in P{q) are trans- 
verse and any two branches of the same periodic point have non-empty inter- 
section. 
Then F will be called Moser generic. 

We remark that for r > 3 the set of Moser generic diffcomorphisms form a 
residual subset of Diff^(S). 

Proposition 8.4. Let F be a Moser generic area preserving diffeomorphism of S'^ 
with an invariant annular domain and let F : A —>■ A be the exension to the prime 
end compactification of this domain. Then we have the following: 

i) the union K. of the irrational invariant essential continua for F is closed; 

ii) the set of the irrational invariant essential continua for F is closed in the 
Hausdorff topology and the function K k^ p{K) is continuous on that set; 
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iii) the connected components of K. are the maximal irrational invariant essential 
continua. 

Proof. We first prove assertion ii). Suppose that the sequence {Kn)n>o of irrational 
invariant essential continua converges to K. This set is an essential invariant con- 
tinuum. Let's suppose that it is not irrational, ft contains an essential hyperbolic 
periodic point z, an essential simple closed curve C C VF*(z) U W^{z), a point 
above that curve and a point below. For n big enough, Kn also contains a point in 
each component of the complement of C . Wc deduce that Kn meets the curve and 
hence must contain the point z. We have a contradiction to the assumption that 
Kn is irrational. 

Let's suppose that the sequence (an)n>o converges to a, where a„ is the rotation 
number of Kn- We can choose for any n > an invariant measure /i„ G M{Kn)- 
Any limit point /x of the sequence {^n)n>o belongs to M{K) and its rotation number 
is p{K). It is also equal to a. We deduce easily the continuity of the map K h^ p{K) 
from this argument. 

Assertion i) is straightforward. If (z„)„>o is a sequence in }C, we can choose for 
every n > an irrational invariant essential continuum Kn which contains Zn- Any 
limit point of the sequence {Kn)n>o is an irrational invariant essential continuum 
which contains z. 

Assertion iii) is also very simple. Any connected component of /C is closed, 
invariant, essential and does not contain any periodic orbit. So it is irrational. Of 
course, it is maximal among the irrational invariant essential continua. D 

The complement of JC is an open set U which contains all the periodic orbits of 
F. Every connected component oiU is an invariant essential open annulus. 

Definition 8.5. Suppose F is a Moser generic area preserving diffeomorphism of 
5^ with an invariant annular domain Aq . Let K, denote the union of the irrational 
invariant essential continua in Aq. Any invariant component of the complement of 
K, in Aq will he called a region of instability. 

Let [/ be a region of instability and U ^U U C+ U C^ the prime end compacti- 
fication of U , where C+ is the upper boundary circle and C^ the lower one. Let's 
write F for the extension of F\ij to U . We know that the rotation numbers defined 
on C+ and on C~ are irrational. Likewise, in S^ wc will denote by 9+[/ and d^U 
the frontiers of the two components of the complement of U . 

Theorem 8.6. Let F be a Moser generic area preserving diffeomorphism of S*^ 
with an invariant annular domain Aq and let U be a region of instability in Aq. We 
have the following results: 

i) The rotation set of U is a segment p{U) of length > 0. 

ii) For every rational number p G p{U), there exists an essential hyperbolic pe- 
riodic orbit of type (p, q) in U , where p ~ p/q and p and q are relatively 
prime. 

iii) Let X = K{z) denote the closure in U of a branch of an essential hyperbolic 
periodic point z. Then X is independent of the essential hyperbolic periodic 
point z and of the branch. It is a subset of any invariant essential continuum 
contained in U which is not a subset of d^U or d~U . 

iv) The set X = K{z) defined to he the closure in U of a branch of z is inde- 
pendent of the essential hyperbolic periodic point z and of the branch. It is a 
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subset of any invariant essential continuum contained in U which is distinct 
from C+ and from C^ . 

Proof. First we prove assertions i) and ii). The set U is an invariant essential 
continuum which is not included in /C. So it contains a periodic point. This periodic 
point belongs to U because d'^U and d^U are irrational invariant essential continua. 
The rotation numbers of C^ and C+ being irrational, the set p{U) is a segment 
of length > 0. The extension of F to the prime-end compactification of U satisfies 
the hypothesis given at the beginning of Paragraph 7 and we can apply Proposition 
8.1 to the essential continuum U itself: for every rational number p G p{U) there 
exists in U an essential hyperbolic periodic point of type (p, q) , where p = p/q and 
p and q are relatively prime. But this point belongs to U and not to the boundary 
curves. 

We now prove assertion Hi). Let K be an essential invariant continuum contained 
in U and not a subset of d~^UUd~U. The boundary of C/^ is an irrational invariant 
essential continuum. It cannot meet U (since it is irrational) and must be contained 
in dU. More precisely it is contained in d'^U because it is connected and because 
K is not a subset of d~U. In the same way the boundary of U]^ is contained in 
d^U . Hence we conclude that K meets the two boundary components of U . If z is 
an essential hyperbolic periodic point, we can find an essential curve contained in 
W^{z) U VF"(z). The set K meets that curve and must contain K{z). In particular, 
if z' is another essential hyperbolic periodic point contained in U the set K{z') 
contains K{z). 

The proof of the assertion iv) is exactly the same. D 

We deduce immediately the following result about periodic points: 

Corollary 8.7. Let z and z' be two essential hyperbolic periodic points of F. The 
following conditions are equivalent: 

i) the points z and z' are in the same region of instability; 
ii) we have K{z) — K{z'); 
iii) we have VK"(z) n T4^*(z') ^ 0. 

We next consider a result that generalies a property of regions of instability of 
twist maps: 

Theorem 8.8. Let U be a region of instability. For any neighborhood W of d~^U 
in U whose closure doesn't contain d^U, the connected component o/ [ ] F^ (W) 

which contains d^U is equal to d^U . 

Proof. The connected component iiT of || F^^{W) which contains d^U is an in- 

fcez 
variant essential continuum and the boundary of Uf^ is an irrational invariant essen- 
tial continuum. This boundary is connected, doesn't meet U and doesn't contain 
d^U: it is contained in d^U . We deduce that K = 9+C/. D 

We obtain the following corollaries: 

Corollary 8.9. Let U be a region of instability. In any neighborhood W of d^U 
in U there exists a periodic orbit. Also in W there is a homoclinic point of any 
essential periodic point in U . 
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Proof. We write z+ for the upper end of U and z^ for the lower end. Let's consider 
the extension F oi F^jj to the end compactification U. The end z+ is a fixed point of 

F and the sequence ( i(F'^, z^) ) of Lefschctz indices is well defined and constant 

equal to 1. Indeed, the rotation number of C"*" is irrational. Let's consider a closed 
disk D which contains z+ in its interior and which is contained in {W \ dU) U {z^}. 
The connected component of X = || F^{D) which contains z+ is equal to z+ by 

fcGZ 

Theorem 8.8. So, we can find a closed disk D' <Z D which contains z+ in its interior 
and such that dD' DX = 0. We deduce that the set || F^{D'), which is contained 

fcez 
in X, doesn't meet the boundary of D' . Such a set is called an isolating block. To 
such a set we can associate a sequence of integers 




\ \ 0<i<k I I , ~~.^ 

and this sequence is non positive for infinitely many values of k (see [F2] or [LY]). 
The integer Ofc is equal to the sum of the Lefschctz indices (for F*') of all the fixed 
points of F^ whose F orbit lies entirely in D' if there are a finite number of such 
points. For any value k such that a^ < there exists a fixed point of F^ distinct 
from z+, whose orbit lies entirely in D' . 

To prove that there is a homoclinic point in W we consider an arbitrary essential 
fixed point z of F'' and let C be an essential simple closed curve in U consisting of 
segments of W^{z) and W'^{z). We define 

n 

Bn= \J F-\C) 

k— — n 

and let An be the closure of the component of the complement of Bn in U which 
contains 9+C/. We note that F{B„) C Bn+i so An+i C F{An). From this it follows 
that n„>oj4„ C n„>o-F(j4„) = F(n„>o^ra) and hence that X ~ n„>o^n is forward 
invariant under F^^ (and similarly under F). We conclude X is invariant and from 
Theorem 8.8 that X = d~^U and since the family {An} is a nested sequence of 
compact sets we know that An C W for n sufficiently large. The frontier of A„ 
consists of finitely many segments of W^{z) and W'^{z) and hence must contain a 
homoclinic point of 2;. D 

Remark 8.10. In the case where F is a twist map, the Birkhoff theory tells us 
that any irrational invariant essential continuum is the graph of Lipschitz map 
ijj : T^ ^ [0,1]. In particular we don't need the prime end theory. We have a 
natural order on the set of these curves (K < K' if K' is above K) and the function 
K H^ p{K) is strictly monotone. There are some other results that we know in the 
case of a twist map. 

Using variational methods, Mather [M2] proved that there exists a point z E U 
whose a-limit set is contained in d~^U and whose w-limit set is contained in d~U, 
if [/ is a region of instability of a twist map. There exists a topological proof of 
this result [L] which uses the Birkhoff theory for positively and negatively invariant 
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essential continua. We now prove that this result remains valid for generic area 
preserving diffeomorphisms without the requirement of the twist condition. 

Proposition 8.11. Let U be a region of instability. There exists a point x in U 
such that Lu{x) C d^U and a{x) C d^U. 

Proof. If C is an essential simple closed curve in U and if N is the component of its 
complement (in U) which contains d^U, a well-known result of Birkhoff (see [Bi2] 
Chapter VIII, section 8) implies that the component of the compact set 

n p^'w 

fc=0 

containing d^U, must meet C. This component, X, is a positively invariant set 
containing the set d~^U but not equal to it. 

Theorem 8.8 tells us that the globally invariant set r\n>aF"{X) is reduced to 
d~^U. So for any point w G X we have uj{w) C d^U. Theorem 8.8 tells us that 
the connected globally invariant set d(U„>o-F'^"(-^)) meets d~U. If z € U is an 
essential hyperbolic periodic point, Proposition 7.4 says there exists an essential 
closed curve contained in the union of any unstable branch of z and any stable 
branch. If n is large enough, the set F~"(X) must meet this curve. We deduce 
that X meets both unstable branches of z. More precisely, there exists an open 
interval 7 of W^ (z) , which contains z, is disjoint from X, and whose boundary points 
belong to X. This arc is contained in a component V oi S^ \ X and separates V 
into two components (because X is connected) which are connected components of 

s'Miux). 

Similarly we can construct a negatively invariant compact set Y in U U d U , 
containing the set d^U but not equal to it. For any point w gY we have a{y) C 
d^U . Also as before, Y meets both branches of W^{z). 

Consider two points wi,W2 S Y, one in each branch of W''^{z). For large n the 
points F"{wi), F"{w2)Y will be close to z, one in each component of V^ \ 7. Hence 
the connected set F"(F) meets the two components of V\j. The set i^"(F) doesn't 
intersect 7, so it intersects X. Any point of intersection of X and F"{Y) satisfies 
the proposition. D 

Among the periodic orbits that we can find in a region of instability there are 
particular orbits, called well-ordered orbits or Birkhoff orbits. Assertion ii) of The- 
orem 8.6 can be improved: the periodic orbit can be chosen among the Birkhoff 
orbits. There is a natural generalization of Birkhoff orbits in the general case which 
is the notion of unlinked orbit. Boyland [Bo] proved that for a homeomorphism of 
the closed annulus the existence of a periodic orbit of type (p, q) implies the exis- 
tence of an unlinked periodic orbit of the same type. Moreover there is a Lefschetz 
formula for these orbits. With these remarks it is not difficult to improve assertion 
ii) of Theorem 8.6: the periodic orbit can be chosen among the unlinked orbits. 

In the case of a twist map there exists in a region of instability U a well-ordered 
invariant Cantor set of rotation number p for every irrational number in the interior 
of the rotation set of U. Such a set is called an Aubry-Mather set. Using symbolic 
dynamics and the heteroclinic intersections between periodic points of rotation 
number smaller and bigger than p, is it possible to find an invariant (or an unlinked 
invariant) compact set of rotation number p in the general case (see [H]) ? 

The following result is closely related to a result of Boyland [Bo] . 
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Proposition 8.12. Let F be a Moser generic area preserving diffeomorphism of 
S"^ with an invariant annular domain Aq and let K he an essential continuum which 
is the closure of one branch of a hyperbolic fixed point z of F. Let z be a point in 
TT^^{z) in the universal cover A of Aq and let f : A ^ A be the lift which fixes z. 
If K C A is Tr^^{K) and p(z, f) G intp(K, f) then K is the closure of any branch 
ofz, i.e. each branch of z in A is dense in K. 

Proof Let F be a branch of z for the lift /. We can assume without loss of generality 
that it is an unstable branch. Since z is essential there is a fc 7^ such that F inter- 
sects W''{T''{z)) transversely. The A- lemma then implies the closure of F contains 
the closure of VK"(T'^(z)). The A- lemma also implies that the set J consisting of 
all values of k for which F intersects W'^{T''(z)) is closed under addition. 

Another straight-forward A-lemma argument and the fact that the rotation num- 
ber of z is in the interior of the rotation set for K shows that J contains both positive 
and negative elements. From this we may conclude that J is a subgroup of Z, say 
the subgroup dZ. 

If we replace Aq and i^ by a d-fold cover of Aq and a lift of F to it then clearly 
nothing has changed in what we want to prove except that now d has been replaced 
by 1. Le. we may assume without loss of generality that d — 1 or that J = Z. 
Hence we may assume that the closure of F contains the closure of W'^{T'^{z)) for 
every k. 

Now let y be an element of K and let y — 7r(y). The fact that W'^{z, F) is dense 
in K implies that for any e > there is a point x of W^{z, F) which is within e 
of y. Equivalently, there is an integer k and a point x of M^"(T*'(z, /)) which is 
within £ of y. Hence F is dense in K . D 

9. Generic diffeomorphisms of the sphere 

In this section we consider a C^-diffeomorphism F of S"^ which preserves a volume 
form Lo. We will further assume F is Moser generic (see 8.3). 

For every integer g > 1 we write Hyp (i^) (resp. Ellq(i^)) for the set of hyperbolic 
(resp. elliptic) fixed point of F"?. We note 

Hyp(F) - U Hyp,(F) 

the set of hyperbolic (resp. elliptic) periodic points and 

m{F) = U E11,(F) 
<z>i 
the set of elliptic periodic points. 

For every integer q > 1, we note fCq the set of non empty continua which are 
(globally) invariant by F'^ and /C* C K,q the set of continua which do not consist of 
a single fixed point of F''. Wc note /C = Ug>i/Cq the sets of periodic continua and 
K,* = yjq>iK,* the set of non trivial periodic continua. 

We recall a property we will use in this section - that a closed set K d S^ \s 
connected if and only if the complement U ~ S'^ \K is simply connected. 

From our earlier results we have the two following properties 

Proposition 9.1. Any continuum K Cz K,* which contains a hyperbolic periodic 
point z contains the four branches of z. 
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Proposition 9.2. Any continuum K G ]Cq which doesn't contain a periodic point 
separates the sphere in two connected and simply connected components which are 
F"^ -invariant. Moreover the rotation set of the continuum consists of a single irra- 
tional number (defined up to sign). 

A periodic continuum without any periodic orbit will be called a periodic irra- 
tional continuum and we will define its rotation number which is an element of T^ 
defined up to sign. 

Let K and K' be two non disjoint periodic irrational continua and write U and 
V for the components oi S^\K and U' and V for the components of S'^\K'. The 
set K U K' is a periodic irrational continuum and the set 

s^\{Ku K') = {ur\ u') u (t/ n V) u (y n u') u [v' n u') 

has exactly two connected components by 9.2. Each set U and V contains a periodic 
point and cannot be contained in K': we deduce that one of the sets t/ fl C/' or 
U C\V' is non empty as is one of the sets V fMJ' and V {^V . Doing the same 
thing for U' and V we can suppose (exchanging U and V if necessary) that the 
two components oi S'^\K\JK' aieU nU' and VnV and ihatU nV ^VfMJ' ^ %. 
The sets U and U' contain the same periodic points as do the sets V and V' . The 
open sets U UU' and V UV are connected and simply connected, because U OU' 
and V OV are non-empty and connected. Thus the sets UUU' and VUV are the 
components of the complement of the irrational continuum K (1 K' . 

By induction we can prove the analogous result for more than two periodic ir- 
rational continua. More precisely, if {Ki)i<i<n is a family of periodic irrational 
continua such that the union is connected, the union is a periodic irrational con- 
tinuum. It's complement has two components ni<i<„C/i and ni<i<„\^ where Ui 
and Vi are the components (well chosen) of S"^ \ Ki. The set r\i<i<nKi is a peri- 
odic irrational continua and the components of its complement are Ui<i<„t/i and 

Ul<i<nK- 

Let K he a periodic irrational continuum whose interior O is not empty and 
consider the two components U and V oi S^\ K . The periodic irrational continua 
dU and dV don't intersect. This is because the sets U , V and O are in three 
separate components of S^ \ {dU U dV) and if dU meets dV their union is an 
invariant irrational continuum whose complement has only two components. Thus 
the set O must be an open annulus. 

Let r^ be a stable branch of z e Hyp (i^) and F" an unstable one. Then, 
the two branches intersect, they have the same closure K{z) S ICq and there is 
a fundamental system of neighborhoods of z which are Jordan domains with a 
boundary contained in F* U F". We know that the equality K{z) = K{z') occurs if 
and only if any stable branch of z meets any unstable branch of z' . We know that 
there is no strict inclusion K{z) C K{z'). 

Let's prove the following fundamental result. 

Proposition 9.3. If {Ui)ii^i is a family of periodic simply connected open sets, the 
union U ~ Ui,=iUi is simply connected. 

Proof. It is sufficient to prove this result in the case where / is finite and the Ui are 
connected. In fact, appealing to induction, it is sufficient to prove that the union 
of two invariant simply connected domains U and U' is simply connected. 

The proof is obvious if one of the complements S^ — U or S^ — U' is empty or 
reduced to a point. We will suppose that none of these properties occurs. The 
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boundaries dU and dU' arc irrational invariant continua and we can write 

S^ = UUdUUV = U'udU' UV' 

where V and V' are some invariant simply connected domains. 

If the sets dU and dU' meet, their union is an irrational invariant continuum 
and the complement 

s"^ \ {dU u du') ^{un u') u (t/ n V) u (y n u') u (y n V) 

has exactly two components, by 9.2. At least one of the sets V f\U' or V \^V' \s 
not empty (because dU' has no interior and cannot contain V). We deduce that 
f7 n [/' is either empty or connected and that [/ U C/' is simply connected. 

If dU is contained in f/', then one of the open sets U ox V doesn't meet the 
connected set 5^ \ U' and is contained in U' . In the first case t/U t/' = C/' is simply 
connected, in the second case we have 

s'^\u = duyjv (zV 

soU iJU' — U \s simply connected. 

If dU is contained in V' , then one of the open sets U ot V doesn't meet the 
connected set S^ \ V and is contained in V. In the first case U and U' are disjoint 
and [/ U [/' is simply connected, in the second case we have 

S^\U = dUUV cV cS^\U' 

and [/ U [/' = S*^ is simply connected. D 

Corollary 9.4. Let U and U' be two disjoint periodic simply connected domains. 
If {Ui)i(£i is the family of all periodic domains disjoint from U and U' , then the 
complement K(U, U') of 

UUU' U,ei U, 

is the smallest periodic continuum such that U and U' are components of its com- 
plement. Moreover K{U, U') is invariant if U and U' are invariant. 

Proof. By Proposition 9.3, we know that K(U, U') is connected. It is clear that U 
and U' are connected components of S"^ \ K{U, U'). If X is a periodic continuum 
such that U and U' are components of S'^ \ K, then the other components are in 
the family {Ui)i(zj, so K contains K{U,U'). To see that K{U,U') is invariant if 
U and U' are we observe that Ui^iUi is i^-invariant in that case. We still have to 
prove that K is periodic, but this follows by considering F'' where g is a common 
period for U and U' and noting that K{U, U') is the same for F and F'', so our 
previous remark implies K{U, U') = F''{K{U, U')). D 

Proposition 9.5. Let U and U' he two disjoint periodic simply connected domains. 
If S''^\(UUU') contains a periodic point, so does K{U, U'). Moreover every periodic 
point z G K(U, U') is hyperbolic. 

Proof. We can suppose that U and U' are invariant. If K{U^ U') has no periodic 
point it is an irrational continuum, so the components of K(U, U') are exactly U 
and U' and we deduce that K{U, U') =S^\{UU U'). 

Let's prove now the second part of the proposition. There is no periodic point 
in dU and in dU' because these two sets are periodic irrational continua. So every 
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elliptic periodic point z E S'^ \{U U U') has a periodic simply connected neighbor- 
hood, bounded by a Jordan curve, which meets neither U nor U' . We deduce that 
z^K{U,U'). U 

We can improve this result. 

Proposition 9.6. Let U and U' be two disjoint invariant simply connected do- 
mains. Consider a fixed point z eU, a fixed point z' G V and a lift f of F\g2\f^.,n 
to the universal covering space of the annulus S^ \ {z, z'}. Let p G Z and q > 1 
relatively prime. Lf S'^ \ {U U U') contains a periodic point of type (p, q), so does 

K{U,U'). 

Proof. The set Z of periodic points of type (p, q) in S*^ \ (f7 U U') is finite. If it is 
disjoint from K{U, U'), there exists a smallest finite family (C/i)i<i<„ of components 
of 5*^ \ K{U, U') which covers Z. Every Ui is i^'^-invariant and every fixed point of 
F'^ in Ui belongs to Z. The contradiction comes from the Lefschetz formula 



O^Y.'^F\z)^Y.'(F^^U,) 



n. 
zez i=i 

D 

We can be more precise. 

Proposition 9.7. Let U and U' two disjoint invariant simply connected domains. 
For any periodic hyperbolic point z E K{U,U'), the set K{z) (defined to be the 
closure of a branch of z) is invariant and separates U and U' . Moreover if T^ is a 
stable branch of z and F" and unstable one, there is a Jordan curve which separates 
U and U' and which is contained in F" U F". 

Proof. The periodic continuum K{z) is contained in K{U, U'). If it doesn't separate 
U and [/', the component oi S'^\K(z) which contains U and U' is a periodic simply 
connected domain V and z ^V since z has a fundamental system of neighborhoods 
whose boundaries are in the branches of z. We deduce that the component of 5*^ \ I^ 
which contains z is a periodic domain disjoint from U and U' . It is disjoint from 
K{U, U') by definition of this set so we have a contradiction. 

Let's prove now that K{z) is invariant by showing that the orbit of z is contained 
in K{z). Consider the components V and V of S'^\K{z) which contain respectively 
U and U' . The set dV is an _F'-invariant irrational continuum and meets its 
image because it separates U and U' and F is area preserving. We deduce that 
K ~ lJo<k<qF''{dV) is an i^- invariant irrational continuum. The orbit of z doesn't 
meet the component r\o<k<qF''{V) oi S"^ \ K which contains U, it is contained in 
the other component which is disjoint from V. Similarly the orbit doesn't meet 
V' . It doesn't meet any other component of 5*^ \ K{z) because it is contained in 
K{U, U'). We have proven that it is contained in K{z). 

We deduce that any stable branch of z' in the orbit of z meets any unstable 
branch of z. We deduce from this fact and from the A- lemma the end of the 
proposition in the case where z is a periodic point of period q > 2. But if z is a 
fixed point then we choose a hyperbolic periodic point w E K(z) of period at least 
two and observe that K{z) = K{w). The fact that W^*(w) intersects F" transversely 
and W'^{w) intersects F* transversely allows us to apply the A- lemma once again 
and obtain a simple Jordan curve separating U and U' which is a subset of F" U F" . 

D 
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Corollary 9.8. Any non-trivial invariant continuum K which is not irrational 
contains an invariant continuum K{z), for some z G Hyp(_F). 

Proof. If the complement of K has at least two invariant domains, this is a con- 
sequence of Propositions 9.5 and 9.7. In the other case the Lefschetz formula tells 
us that there exists at least one fixed point in K. If K contains a hyperbolic fixed 
point, the result is obvious. If K contains an elliptic fixed point z, we can find 
two minimal invariant curves Ci and C2 of different irrational rotation number 
surrounding z in a neighborhood of that point. These curves meet K and are 
contained in it. By the Poincare-Birkhoff theorem, we can apply Proposition 9.5 
and 9.7 to the domain U containing z bounded by the closest curve and to the 
domain U' which doesn't contain z and bounded by the other curve. There exists 
z' G Hyp(F), such that K{z') is invariant and a simple curve separating U and U' 
and contained in W"{z') U W^*(z'). This curve meets the connected set K and we 
have K(z') <zK. U 

Corollary 9.9. The invariant continua K{z), z G Hyp(i^), are the minimal ele- 
ments in the set of the non trivial invariant continua which are not irrational. 

Proof. This is an immediate consequence of Corollary 9.8 and of the fact that there 
is no strict inclusion K{z) C K{z')^ for z, z' G Hyp(F). D 

Proposition 9.10. The set of irrational invariant continua and continua {z}, z G 
Elli(_F), is closed in the Hausdorff topology. The union of all these sets is closed. 

Proof Recall that the set ICi consists of invariant continua. The result is a conse- 
quence of Corollary 9.8 and of the following facts: 

• the set /Ci is closed in the Hausdorff topology; 

• any continuum {z}, z G Hyp]^(_F), is isolated in /Ci; 

• for any z G IIyp(_F), the set of continua K E JCi which contain K(z) is open 
in /Ci. 

D 

Remark 9.11. By Birkhoff's theory, if z is a elliptic fixed point and V a small 
neighborhood of z, there is a neighborhood of {z} in the set defined in Proposition 
9.10 which consists of {z} and of the invariant curves surrounding z and contained 
in V . If z' is another fixed point, the set which consists of {z}, {z'} and of the 
irrational invariant continua which separate z and z' is closed. 

Proposition 9.12. The set consisting of irrational invariant continua, of invari- 
ant sets K{z), for z G IIyp(_F), and of continua {z}, z G Elli(_F), is closed in the 
Hausdorff topology. 

Proof. Indeed, any convergent sequence K{zn), z„ G Hyp(F), of invariant continua 
converges either to a trivial continuum {z}, z G Elli(_F), or to an irrational invariant 
continuum, or to an invariant continuum which contains an invariant set K{z), 
z G Ilyp(i^). In the last situation the set K{z) is contained in _?C(z„), for n > uq 
big enough. The sequence K{zn)n>no i^ constant and equal to K{z). D 

A periodic domain U of period <? > I will be called a strictly periodic domain 
if the domains F^{U), < k < q are all disjoint. Note that the closures also 
are disjoint. Indeed if dU meets dU' where U' = F^{U), < k < q, the union 
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of these two sets is an i^'^-invariant irrational continuum. Writing V = S'^ \ U 
and V' ~ S'^ \U', the components of the complement must he U OV ~ U and 
U' OV = U' (because U and U' are disjoint) and we must have y n V = 0. This 
is possible only if g = 2, and F{U) = U', F{U') = t/ and 5^ = [7 U 17^. But that 
leads to a contradiction since there are no fixed point in [/ U t/' there must (by the 
Lefschetz theorem) be a fixed point in dU U dU' . 

Proposition 9.13. The union Xi of the irrational invariant continua, of the in- 
variant sets K{z), z G Hyp(_F) and of the elliptic fixed points is an invariant 
continuum. Its complement is the union of the simply connected strictly periodic 
domains of period q > 2 

Proof. The fact that it is closed is a consequence of Proposition 9.12. We prove that 
it is connected by showing that any component U oi S'^ \ Xi is simply connected. 

We first observe that U is not invariant. If it were invariant, its closure would be 
an invariant continuum which is not irrational (because U (t Xi). It would contain 
an invariant continuum K{z), z G Hyp(_F), which is disjoint from U (because 
U Xi ~ (d). A contradiction arises from the fact that there is a fundamental 
system of neighborhoods of z whose boundary is in W^{z) U W"{z). 

Next we prove that U is simply connected. Any Jordan curve 7 contained in 
U is disjoint from its image. The map F being volume-preserving, one of the two 
domains bounded by 7 is disjoint from its image. Any component of Xi being 
invariant, we deduce that this domain is disjoint from Xi and contained in U. 
We conclude that U is simply connected. We have proven that any component of 
S'^ \ Xi is a strictly periodic simply connected domain of period q > 2. 

Conversely, a strictly periodic domain U of period q > 2 doesn't contain any fixed 
points. It cannot meet any fixed continuum K{z), z G Hyp(_F) since otherwise it 
will meet 14^" (z) and the closure U which is disjoint from its image will contain 
K{z). 

Finally we must show that U cannot meet any irrational invariant continuum 
K. If it did then we could not have K C U so K D dU 7^ 0. It follows that dU is a 
periodic irrational continuum and by our remarks above that K' = KUdUUF{dU) 
is a periodic irrational continuum. Since K has no interior, the complement of K' 
would necessarily have at least 3 components since it would contain a point from 
U, a point from F(U) and a point in neither of them. Each of these would be in 
different components of S*^ \K' ■ This contradicts 9.2. D 

We will write E for the union of the periodic irrational continua and of the 
elliptic periodic points, H for the union of the K{z), z G Ilyp(i^) and / for the 
complement oi E U H. So we have a decomposition 

S^ = EUHUL 

Proposition 9.14. We have the following properties : 

1) the sets E and H are F„ ; 
ii) we have I C dE — dH ; 
iii) the set E U H is connected and dense ; 
iv) a point z belongs to I if and only if there exists a decreasing sequence (lJn)n>a 

of strictly periodic simply connected domains which contain z such that the 

sequence of periods tends to the infinity. 
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Proof. We can write E — Uq>iEq and H = (Jq>iHq, where Eq is the union of 
the _F'''-invariant irrational continua and of the eUiptic fixed points of F"^ and Hq 
the union of the sets K{z), z € Hyp(i^), which are F'^-invariant. Each set Eg 
is closed and each set Hq is an F^, moreover Eq U Hq is closed, connected and 
contains Hi U Ei. The first assertion is clear as is the connectedness oi E U H. 
The last assertion is an immediate consequence of Proposition 9.13. The density of 
EUH follows from the inclusion I <Z E that we will prove now. For any decreasing 
sequence {Un)n>o of strictly periodic domains which contains z G I such that the 
sequence of periods tends to infinity, the volume of [/„ tend to zero. So dUn which 
is contained in E meets a given neighborhood of z if n is big enough. 

Since I C E we deduce that dH C dE. If we show I C H then it follows that 
dE C OH and we can conclude assertion ii). If zq is in / then by iv) z € U where 
U is a strictly periodic simply connected domain of high period (and hence of small 
measure) . Given e > we may choose U so that any point of U is within e of dU. 
We also know that z G U' C U where U' is a strictly periodic simply connected 
domain of higher period than U. The annular domain U\U is invariant under some 
iterate of F and contains periodic points, so we may apply Proposition 8.1 to its 
prime end compactification to conclude that it has an essential periodic point zq. 
Then K{zo) separates z and dU. It follows that there must be a point of K{zo) C H 
within e of z and hence I C H. D 

Theorem 9.15. There exists a dense Gs, Q in Diff^(S'^) such that the interior of 
E is empty if F G Q . The union of stable manifolds of hyperbolic periodic points is 
dense in S"^ for F in Q . 

Proof. It is sufficient to prove that among the generic diffeomorphisms studied in 
this section (i.e. the Moser generic diffeomorphisms, see 8.3), the ones which don't 
have any periodic open annulus without periodic points is generic. For such a map 
the interior of the sets Eq, q > \, \s empty as is the interior of the union E. We 
deduce the second part of the proposition from the inclusion / U dE C dH . 

Thus it suffices to prove that there is a generic set of diffeomorphisms in Diff ^ {S'^ ) 
which have no periodic open annulus which contains no periodic points. Consider 
an open connected and simply connected subset V C S^ . We will first show that 
for a dense open subset of Diff^(S'^) the set V is not contained in a periodic open 
annulus which contains no periodic points. 

Consider the set of diffeomorphisms Qv C Diff^(S'^) defined by the property 
that F G Qv if a-nd only if there are two disjoint simple closed curves Ci and 
C2 contained in one component of Unez-^"(^) ''^ith the property that there are 
hyperbolic periodic points of F in each of the three components of S"^ \ (Ci U C2). 
It is clear that ii F G Qv then V is not contained in a periodic open annulus which 
contains no periodic points since Ci and C2 would have to be contained in such an 
annulus and this is impossible. We will show that Qv is dense and open. 

The fact that Qv is open is clear since for any diffeomorphism G close to F we 
know Ci U C2 is contained in one component of IJnez G"(V) because Ci U C2 is 
compact. Also if G is close to F it will still have hyperbolic periodic points in the 
three components of S^ \ (Ci U C2). So Qv is open. 

To see that Qv is dense consider any Fq E Diff^(S'^). We can approximate 
it arbitrarily closely by a Moser generic diffeomorphism F. We will show either 
F G Qv or it can be approximated by an element oiQv- Consider U = [J^ez ^"(^)- 
If U contains a periodic point of F then it contains infinitely many hyperbolic 
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periodic points since for Moser generic diffeomorphisms hyperbolic periodic points 
are not isolated and elliptic periodic points are in the closure of hyperbolic periodic 
points. It is thus clear that if U contains any periodic points we may choose simple 
closed curves Ci and C2 contained in one component of U such that each component 
of 5^ \ (Ci U C2) contains a hyperbolic periodic point. Thus in this case F ^ Qv- 

So we may assume U contains no periodic points. We choose a periodic compo- 
nent Uq of [/, say of period g, so F'^{Uo) = Uq- U the complement of Uq has three 
or more components containing periodic points we can, as before, choose Ci and 
C2 showing F £ Qy. It is not possible for the complement of Uq to have only one 
component containing periodic points, since the complement of that component 
would be a simply connected F"^ invariant domain with no periodic points. 

All that remains is the case that the complement of Uq has exactly two compo- 
nents containing periodic points. We choose a simple closed curve C in Uq which 
separates these two components. The complement of these two components is an 
open f invariant annulus W containing Uq and in which C is essential. 

We choose a small annular neighborhood Ui of C in Uq- Perturbing F by the 
time t map of an area preserving C°° flow (j)t supported in the interior of Ui we may 
continuously vary the mean rotation number of F'^ o (pi = {^F o (ptY on the annulus 
W . However a result of [F3] says that if this number is rational there must exist 
a periodic point. Clearly if t > is small the perturbation is C close to F. One 
point on the new periodic orbit must lie in the interior of C/i since that is where 
the perturbation is supported. By a slight further perturbation we obtain G close 
to F with a hyperbolic periodic point in the interior of Ui . Choosing Ci and C2 to 
be the boundary components of Ui we observe that G <^Qv- 

Thus we have shown that Qv is open and dense. If we now choose a countable 
basis {Vn} for the topology of S"^ consisting of open disks and let 



n^v 



is the desired dense Gs- □ 
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